Colour Helmholtz Stereopsis for Reconstruction of Complex Dynamic Scenes

Helmholtz Stereopsis (HS) is a powerful technique for reconstruction of scenes with arbitrary reflectance properties. However, previous formulations have been limited to static objects due to the requirement to sequentially capture reciprocal image pairs (i.e. two images with the camera and light source positions mutually interchanged). In this paper, we propose colour HS-a novel variant of the technique based on wavelength multiplexing. To address the new set of challenges introduced by multispectral data acquisition, the proposed novel pipeline for colour HS uniquely combines a tailored photometric calibration for multiple camera/light source pairs, a novel procedure for surface chromaticity calibration and the state-of-the-art Bayesian HS suitable for reconstruction from a minimal number of reciprocal pairs. Experimental results including quantitative and qualitative evaluation demonstrate that the method is suitable for flexible (single-shot) reconstruction of static scenes and reconstruction of dynamic scenes with complex surface reflectance properties

A Bayesian Framework for Enhanced Geometric Reconstruction of Complex Objects by Helmholtz Stereopsis

Helmholtz stereopsis is an advanced 3D reconstruction technique for objects with arbitrary reflectance properties that uniquely characterises surface points by both depth and normal. Traditionally, in Helmholtz stereopsis consistency of depth and normal estimates is assumed rather than explicitly enforced. Furthermore, conventional Helmholtz stereopsis performs maximum likelihood depth estimation without neighbourhood consideration. In this paper, we demonstrate that reconstruction accuracy of Helmholtz stereopsis can be greatly enhanced by formulating depth estimation as a Bayesian maximum a posteriori probability problem. In reformulating the problem we introduce neighbourhood support by formulating and comparing three priors: a depth-based, a normal-based and a novel depth-normal consistency enforcing one. Relative performance evaluation of the three priors against standard maximum likelihood Helmholtz stereopsis is performed on both real and synthetic data to facilitate both qualitative and quantitative assessment of reconstruction accuracy. Observed superior performance of our depth-normal consistency prior indicates a previously unexplored advantage in joint optimisation of depth and normal estimates

Accurate 3D reconstruction of dynamic scenes with complex reflectance properties.

Accurate 3D geometry modelling is an essential technology for many practical applications (computer generated imagery, assisted surgery, heritage preservation, automated quality control, robotics etc.). While the existing reconstruction methods mainly operate assuming the simplistic Lambertian model, real scenes, static or dynamic, are characterised by arbitrarily complex a priori unknown reflectance properties. The reflectance limitation of the state-of-the-art causes a gap between the practical demand for photometrically arbitrary scene modelling and the constrained applicability scope of existing methods. In response to the gap, this dissertation proposes a solution to the challenging problem of accurate geometric reconstruction of dynamic scenes with arbitrary a priori unknown reflectance. This is achieved by introducing a novel approach which generalises Helmholtz Stereopsis (HS) - a niche technique known to be independent of surface reflectance but till now limited to static scenes requiring sequential acquisition of a large number of input views. The undertaken generalisation extends the technique to dynamic scenes by two mutually tailored developments in response to the shortcomings of conventional HS. These developments are 1) a framework to fundamentally improve the geometric reconstruction accuracy from a small set of input images and 2) the design of a novel wavelength-multiplexing-based pipeline for dynamic scene modelling. Together these constitute a novel practical system which, for the first time, enables reconstruction of dynamic scenes with arbitrary surface properties. To improve the quality of geometric reconstruction by HS, a novel Bayesian formulation of the technique is proposed to replace its sub-optimal maximum likelihood formulation. Further a tailored prior enforcing consistency of per-point depth and normal estimates and related to integrability is developed. The prior purposely exploits the unique ability of HS to characterise the surface by both estimates. The formulation embedded into a coarse-to-fine framework without explicit surface integration achieves unprecedented accuracy and resolution of geometric modelling by HS regardless of reflectance, competitive with what the non-HS state-of-the-art achieves with strictly constrained reflectance. To generalise HS to dynamic scenes, Colour Helmholtz Stereopsis (CL HS) is proposed which utilises wavelength multiplexing for simultaneous acquisition of the minimal set of input images required for reconstruction. The challenges imposed by wavelength multiplexing in CL HS are addressed using a specially designed calibration consisting of two mutually dependent parts: one infers the photometric properties of the acquisition equipment while the other estimates the reconstructed surface chromaticity spatially and propagates it temporally to accommodate dynamic surface deformation. By integrating the proposed coarse-to-fine Bayesian HS with integrability prior into CL HS, remarkable accuracy and resolution of reconstruction are achieved with the minimal input using just three RGB cameras. Evaluation validates the approach by reconstruction of dynamic scenes with arbitrary a priori unknown reflectance, which includes unconstrained spatially varying chromaticity. The reconstructed dynamic sequences exhibit high per-frame geometric accuracy and resolution as well as temporal consistency

Accurate 3D reconstruction of dynamic scenes with complex reflectance properties.

Accurate 3D geometry modelling is an essential technology for many practical applications (computer generated imagery, assisted surgery, heritage preservation, automated quality control, robotics etc.). While the existing reconstruction methods mainly operate assuming the simplistic Lambertian model, real scenes, static or dynamic, are characterised by arbitrarily complex a priori unknown reflectance properties. The reflectance limitation of the state-of-the-art causes a gap between the practical demand for photometrically arbitrary scene modelling and the constrained applicability scope of existing methods. In response to the gap, this dissertation proposes a solution to the challenging problem of accurate geometric reconstruction of dynamic scenes with arbitrary a priori unknown reflectance. This is achieved by introducing a novel approach which generalises Helmholtz Stereopsis (HS) - a niche technique known to be independent of surface reflectance but till now limited to static scenes requiring sequential acquisition of a large number of input views. The undertaken generalisation extends the technique to dynamic scenes by two mutually tailored developments in response to the shortcomings of conventional HS. These developments are 1) a framework to fundamentally improve the geometric reconstruction accuracy from a small set of input images and 2) the design of a novel wavelength-multiplexing-based pipeline for dynamic scene modelling. Together these constitute a novel practical system which, for the first time, enables reconstruction of dynamic scenes with arbitrary surface properties. To improve the quality of geometric reconstruction by HS, a novel Bayesian formulation of the technique is proposed to replace its sub-optimal maximum likelihood formulation. Further a tailored prior enforcing consistency of per-point depth and normal estimates and related to integrability is developed. The prior purposely exploits the unique ability of HS to characterise the surface by both estimates. The formulation embedded into a coarse-to-fine framework without explicit surface integration achieves unprecedented accuracy and resolution of geometric modelling by HS regardless of reflectance, competitive with what the non-HS state-of-the-art achieves with strictly constrained reflectance. To generalise HS to dynamic scenes, Colour Helmholtz Stereopsis (CL HS) is proposed which utilises wavelength multiplexing for simultaneous acquisition of the minimal set of input images required for reconstruction. The challenges imposed by wavelength multiplexing in CL HS are addressed using a specially designed calibration consisting of two mutually dependent parts: one infers the photometric properties of the acquisition equipment while the other estimates the reconstructed surface chromaticity spatially and propagates it temporally to accommodate dynamic surface deformation. By integrating the proposed coarse-to-fine Bayesian HS with integrability prior into CL HS, remarkable accuracy and resolution of reconstruction are achieved with the minimal input using just three RGB cameras. Evaluation validates the approach by reconstruction of dynamic scenes with arbitrary a priori unknown reflectance, which includes unconstrained spatially varying chromaticity. The reconstructed dynamic sequences exhibit high per-frame geometric accuracy and resolution as well as temporal consistency

Bayesian Helmholtz Stereopsis with Integrability Prior

Helmholtz Stereopsis is a 3D reconstruction method uniquely independent of surface reflectance. Yet, its sub-optimal maximum likelihood formulation with drift-prone normal integration limits performance. Via three contributions this paper presents a complete novel pipeline for Helmholtz Stereopsis. Firstly, we propose a Bayesian formulation replacing the maximum likelihood problem by a maximum a posteriori one. Secondly, a tailored prior enforcing consistency between depth and normal estimates via a novel metric related to optimal surface integrability is proposed. Thirdly, explicit surface integration is eliminated by taking advantage of the accuracy of prior and high resolution of the coarse-to-fine approach. The pipeline is validated quantitatively and qualitatively against alternative formulations, reaching sub-millimetre accuracy and coping with complex geometry and reflectance

Colour Helmholtz Stereopsis for Reconstruction of Dynamic Scenes with Arbitrary Unknown Reflectance

Helmholtz Stereopsis is a powerful technique for reconstruction of scenes with arbitrary re ectance properties. However, previous formulations have been limited to static objects due to the requirement to se- quentially capture reciprocal image pairs (i.e. two im- ages with the camera and light source positions mu- tually interchanged). In this paper, we propose Colour Helmholtz Stereopsis - a novel framework for Helmholtz Stereopsis based on wavelength multiplexing. To ad- dress the new set of challenges introduced by multispec- tral data acquisition, the proposed Colour Helmholtz Stereopsis pipeline uniquely combines a tailored pho- tometric calibration for multiple camera/light source pairs, a novel procedure for spatio-temporal surface chromaticity calibration and a state-of-the-art Bayesian formulation necessary for accurate reconstruction from a minimal number of reciprocal pairs. In this frame- work, re ectance is spatially unconstrained both in terms of its chromaticity and the directional component dependent on the illumination incidence and viewing angles. The proposed approach for the rst time en- ables modelling of dynamic scenes with arbitrary un- known and spatially varying re ectance using a practi- cal acquisition set-up consisting of a small number of cameras and light sources. Experimental results demon- strate the accuracy and exibility of the technique on a variety of static and dynamic scenes with arbitrary un- known BRDF and chromaticity ranging from uniform to arbitrary and spatially varying

Helmholtz Stereopsis Synthetic Dataset

The dataset consists of synthetic images for three test objects intended to be used as a benchmark for reconstruction via Helmholtz Stereopsis. This also includes ground truth data for quantitative evaluation

Colour Helmholtz Stereopsis for Reconstruction of Dynamic Scenes with Arbitrary Unknown Reflectance

Helmholtz Stereopsis is a powerful technique for reconstruction of scenes with arbitrary re ectance properties. However, previous formulations have been limited to static objects due to the requirement to se- quentially capture reciprocal image pairs (i.e. two im- ages with the camera and light source positions mu- tually interchanged). In this paper, we propose Colour Helmholtz Stereopsis - a novel framework for Helmholtz Stereopsis based on wavelength multiplexing. To ad- dress the new set of challenges introduced by multispec- tral data acquisition, the proposed Colour Helmholtz Stereopsis pipeline uniquely combines a tailored pho- tometric calibration for multiple camera/light source pairs, a novel procedure for spatio-temporal surface chromaticity calibration and a state-of-the-art Bayesian formulation necessary for accurate reconstruction from a minimal number of reciprocal pairs. In this frame- work, re ectance is spatially unconstrained both in terms of its chromaticity and the directional component dependent on the illumination incidence and viewing angles. The proposed approach for the rst time en- ables modelling of dynamic scenes with arbitrary un- known and spatially varying re ectance using a practi- cal acquisition set-up consisting of a small number of cameras and light sources. Experimental results demon- strate the accuracy and exibility of the technique on a variety of static and dynamic scenes with arbitrary un- known BRDF and chromaticity ranging from uniform to arbitrary and spatially varying